feat: retry and redrive dead effects and broadcasts - #78

Closes #73.

Why

DeadLetterManager covered message dead letters only. A dead effect or
broadcast reached status = 'dead' through the same check constraint and
nothing brought it back, so the recovery path was an operator writing an
UPDATE against a runtime table. That is the one thing the deny-by-default
posture exists to prevent, and for a transmit effect it meant a dead one was a
lost replay.

Retry was also one row at a time. An incident produces dead rows in the
hundreds.

The kind rides on the receiver

SolidObjects.dead_letters              # messages, unchanged
SolidObjects.dead_letters.effects
SolidObjects.dead_letters.broadcasts

SolidObjects.dead_letters.all and .retry(id) behave exactly as they did.
Each scope reads and retries only its own kind and authorizes under its own
resource name: effect_dead_letters, broadcast_dead_letters, redrives.

Retry

SolidObjects.dead_letters.effects.retry(effect_id, authorization_context: current_admin)

It returns the row to pending with a zero attempt count, no claim, and immediate
availability, and it keeps the stable id so a handler that deduplicates on
effect_id still sees the same key.

Two decisions worth review, because they are mine rather than the issue's.

Retry acts only on a dead row. A row that is pending, processing, or
completed comes back unchanged. That is what makes a second press a no-op, and
it also means a retry can never take a row away from a worker that holds it.

Retry takes the stable id, effect_id or broadcast_id, not the primary
key. The issue names the parameters that way, and it is the id a deduplicating
handler keys on. Message dead letters still take the row id, which is their
existing contract.

Redrive

task = SolidObjects.dead_letters.effects.redrive(
  actor_type: "payments", failed_after: 6.hours.ago, limit: 5_000,
  authorization_context: current_admin
)
task.cancel(authorization_context: current_admin)

SolidObjects.redrives.find(task.id, authorization_context: current_admin)
SolidObjects.redrives.all(status: :running, authorization_context: current_admin)

Idempotency is a database constraint, not a read followed by a write. A
redrive writes its scope and filter digest into a unique active_scope
column, so two processes that start the same redrive at the same instant share
one task rather than race. The column holds the digest while the task runs and
NULL once it finishes, which makes the index total rather than partial, since
AGENTS.md rules partial indexes out. The same scope can be redriven again once
the first task ends.

The supervisor advances it. One bounded batch per pass, each its own short
transaction, with redrive_batch_pause between batches, so a redrive of
thousands of rows never holds a transaction longer than one batch and shares the
database with delivery. redrive_batch_size defaults to 100 and
redrive_batch_pause to 0.05 seconds.

A running task reports what is left to move, counted at read time, rather
than a stored estimate. Rows die and are retried while a redrive runs, so a
stored number would drift.

Audit

Every retry and every redrive transition writes one row to
solid_objects_administration_events: action, kind, subject, filters, identity,
and when. The identity comes from the authorization context the caller already
passes, through a new administration_identity hook that defaults to its to_s
and is bounded to 255 bytes. A refused caller writes nothing, because the audit
records what happened rather than what was denied. A read writes nothing.

Schema

Two tables, solid_objects_administration_events and solid_objects_redrives.
No change to effects, broadcasts, or dead_letters, because status
already carries dead and pending. Both are added to the doctor's schema
check and to the public TestHelper reset list, which has a guard test that
fails when a table is missing from it.

Tests

39 new tests across three files, each watched failing first.

test/integration/dead_letter_scopes_test.rb covers the issue's list: a dead
effect returns to pending and runs again, a retried effect reuses its stable id,
retrying a pending effect changes nothing, a dead broadcast returns and
delivers, a dead transmit effect replays, SolidObjects.dead_letters is
unchanged, each scope reads only its own kind and only dead rows, and an
unauthorized caller is refused.

test/integration/redrive_test.rb covers bounded batches, the limit, running
idempotency, a separate task per scope and per filter set, a new task after the
first finishes, cancel leaving moved rows moved, both filters, reading tasks
back, one audit row per transition, refusal, the frozen value object, and a
supervised runtime draining a redrive with no caller driving it.

test/integration/administration_audit_test.rb covers one row per press for all
three kinds, the configured identity, and no row for a refusal or a read.

The behaviours are load-bearing, not just the method names. Removing the batch
cap failed three tests; making the scope digest random failed the idempotency
test with two rows where one was expected.

Validation

bundle exec rake passes, including Standard, RuboCop, RBS, Steep, and
Brakeman.

What this does not do

Automatic redrive on a schedule, which the issue puts out of scope.

The dashboard does not yet surface the new scopes or redrive. The issue
specifies an API and an audit trail, not a UI, and the roadmap entry now says so
rather than implying the dashboard covers it.

🤖 Generated with Claude Code